Robust Learning with FeatureBoost

PAC learning typically assumes that the training and test sets are drawn from the same distributions. This assumption often is violated in practice. How can machine learning algorithms be biased to output hypotheses that are robust to alterations in the test dis-tribution? We create a framework for learning in environments where the test and training distributions diier because features in the test sets are likely to be missing or corrupted. Motivated by this framework, we present a new meta-learning algorithm, Fea-tureBoost. We demonstrate FeatureBoost on three learning problems using backprop nets, k-nearest neighbor, and decision trees. 1 Motivation Consider a backprop net learning to steer a car in Pomerleau's ALVINN system 7]. The principle internal features learned by ALVINN nets detect the left and right edges of the road. ALVINN nets do not learn internal features that detect road centerlines. This creates a problem when the left or right edges of the road are obstructed by passing vehicles, or are missing as on bridges. Yet human steering is remarkably robust to the loss of these features because human drivers can fall back on a number of alternate features as diierent subsets of road features come in and out of view. Backprop nets do learn to steer better if they are forced to learn to recognize centerliness4]. Why do ALVINN nets not automatically learn to use a variety of road features (such as centerlines) when learning to steer? A related problem arises in pneumonia risk predictionn8]. Here there are a number of basic inputs available for patients before they enter the hospital (e.g., age, gender, blood pressure), as well as a number of lab features that become available after patients enter the hospital (e.g., RBC counts, oxygenation, Albumin). Models trained to predict risk from both the pre and in-hospital features are more accurate than models trained from only the pre-hospital inputs, but perform poorly on patients not yet admitted to the hospital (marginalizing over the missing in-hospital features). Models trained to predict risk from only the pre-hospital inputs are more accurate on patients not yet admitted to the hospital than marginalized models trained on all the features. If we can train models to make good predictions from only the pre-hospital inputs, why don't models trained with all the features work this well when some of the features are missing?